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Introduction to Columbia 


Motivation 


• Methodology 



• Concluding remarks 
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Columbia 


• 10240 CPUs 

• 20 SCj^ ‘Afttx superdusters (512 CWS/cfuster) 

• InteC Itanium® 2 processors (1.5 Cj'Hz) 

• iTtB memory/cfuster, 440 r T J ‘B storage-' 

• ijICUK/..-'**- A^iuk : ' b* ,* -i* ? : v . : t .• • I 

1 ii;--! • S^'s 0 'kU'M‘AfFex architecture (InfiniBand) 
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• Top performance to date: 51.9 TFlops 

• Unpack benchmark 


• #2 on the ‘T’OTsoo Cist' 




• Built in <5 months, fully operational 
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r- 1 http://wwv.nas.nasa.gov/About/Projects/Columbia/columbia.html 
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Motivation 


• Need to know if cooling is adequate 

• identify l-Cigh-temperature regions 

• ^Effect ofcahke trays in Hot isfes 


. 

• Find dead-zones 

• SKreas of stagnated air 

j a 

• Short-cycling 

H * Cofd air from fyoor tifes returning directCy to cooCers 
• 1~Cot air from racks returning tv racks 

• Deadline: 6 weeks 
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Methodology 
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• Overset mesh approach 

• ; . » 

• Geometric model 

■ 

• Xfon-CAD / script - based modeling^ 

• r Many structured meshes make up tfie domain of interest' 

• Chimera Ojrid TooCs (CCjT) 
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PDE solution model 

• Compressikke idavier-Stokes equations 

• boundary conditions 

• Overftow2 sofver 
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Body-fitted grids 

• Comptex geometry^ 

Geometry 

• Quadritaterat cells 

• Overtopping^ 

Volume 

’ •/' O-texaHedrat cetts 

• Overtopping^ 
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Overset Mesh Approach 


CPU Racks 
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Simulation on an overset mesh 

1. Generate surface meshes 

2. Generate volume meshes 

3. Specify initial conditions 

4. Apply boundary conditions 

5. Solve on each volume mesh 

6. Interpolate flow data in overlapping regions 

7. Repeat 4-6 until converged (steady results) 
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13 Coolers 


8 Racks + 2 Long 
Racks 


11 Cable Trays 


1 Disk Rack 


Perforated Floor 
Tiles 


Control room, 
window 


Mass Storage 


14 Power^^, 
Distribution Unit 
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Geometric Model 
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PDE Solution Model 
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• Overflow2 (NASA/Army developed) 
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• 102 structured overset mesnes, 12 million points 
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• Solution oj Wavier- Stokes equations 
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• Compressive, viscous jTows 

• J^ovd-speea pre-conditioner jor accuracy \V 
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• Okam steady-state 
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Approach 
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• Develop geometric model 

• 2D sub-floor model 
Develop necessary boundary conditions 


3D main-floor model 
• Flow visualization |j 
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2D sub-floor 



Cooler exit Pipes 
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Perforated tiles. 
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• Simulation indicates that pipes (locking sub-ffoor jTow dove-' 
CittCe effect on the perforated tifeffow rates 

Hie ’ ' _ ffff s hM) '’FfT p*iii ;: ^ 

Xjterature: Tt full three dimensional sub-floor simufation with pipes and 

Conduits found little flow rate variation in floor tiles except near coolers. 

Karki, K.C., Radmehr, A., and Patankar, S.V., “Use of Computational Fluid Dynamics for 
Calculating Flow Rates Through Perforated Tiles in Raised-Floor Data Centers,” 
HVAC&R Research Journal, Vol. 9, No. 2, April 2003, pp. 153-16 
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REASONABLE ASSUMPTION: THE COOLER FLOW RATE 
IS EVENLY DISTRIBUTED AMONG ALL THE 
PERFORATED FLOOR TILES 
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Boundary Conditions 
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• Perforated tiles: Temperature, speed known 

• Rack intake: Speed based on fan rating 
_ , . _ 


• Rack exit: Known temperature increase 

/- -rv’j ‘ ' . V- " - '4. ‘ - . • . 

• Cooler intake: Known speed based on CFM 
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Disk Rack 


Temperature/velocity vectors 
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Flow Down Draft 

Blockage (Streamline Short Cycling) 
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Disk Rack 


Racks 
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Temperature/velocity vectors 















Cooler With 
Diverter 


Cooler Without 
Diverter 


Possible Modification 
Diverter on Cooler 


Sill* 







Used overset mesh technology to evaluate the 
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character of a computer room 

erage temperature in room to 
equacy of the coolers 

;h temperature regions and dead 


Lg of effect of the cable 






. to reduce sfiort-cycCing^ 



